Imaging apparatus, imaging system, and image processing method

ABSTRACT

To increase image quality of a moving image by suppressing a color afterimage while reducing color noise, provided is an imaging apparatus, including: an imaging device; and a signal processing unit, in which: the imaging device includes a first pixel group and a second pixel group each including a plurality of pixels each configured to output a pixel signal; and the signal processing unit is configured to perform weighted addition for a second pixel signal output from the second pixel group by inter-frame processing, and to change a weight on each frame in the weighted addition based on an inter-frame differential of a first pixel signal.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to an imaging apparatus, an imagingsystem, and an image processing method.

Description of the Related Art

In a solid-state imaging apparatus of a single-plate type, in order toobtain a color image, color filters (CFs) each configured to transmitlight of a specific wavelength component, for example, light of a colorof red (R), green (G), or blue (B), are arrayed in a predeterminedpattern. As a pattern of CFs, a pattern having a so-called Bayer arrayis often used. In the following, a pixel in which the CF of R isarranged is referred to as “R pixel”, a pixel in which the CF of G isarranged is referred to as “G pixel”, a pixel in which the CF of B isarranged referred to as “B pixel”, and a pixel in which no CF isarranged is referred to as “W pixel (white pixel or clear pixel)”. Inaddition, the R pixel, the G pixel, and the B pixel are sometimesreferred to collectively as “RGB pixels” or “color pixels”.

In order to improve the sensitivity of the solid-state imagingapparatus, there is proposed a configuration for increasing theproportion of pixels from which information on a luminance is obtainedeasily. Above all, the W pixel that widely transmits light within avisible light range enables to improve the sensitivity and to obtain animage having a high S/N ratio. International Publication No.WO2010/090025A describes an imaging apparatus configured such that aratio of the numbers of RGB pixels and W pixels is R:G:B:W=1:1:1:1. Thisimaging apparatus restores an RGB image for each frame time based onmotion information, which has been detected from an image of W pixels,and images of RGB and an image of W pixels, which have been subjected toaddition for each frame.

The apparatus described in the international publication No.WO2010/090025A is directed to reduce color noise by increasing thenumber of frames of RGB pixels to be subjected to the addition. However,there is a problem in that a color afterimage is caused by an objectexhibiting a large motion.

SUMMARY OF THE INVENTION

According to one embodiment of the present invention, there is providedan imaging apparatus, including: an imaging device; and a signalprocessing uni in which: the imaging device includes a first pixel groupand a second pixel group each including a plurality of pixels eachconfigured to output a pixel signal; the pixel signal output by theplurality of pixels of the second pixel group includes a smaller amountof resolution information than an amount of resolution informationincluded in the pixel signal output by the plurality of pixels of thefirst pixel group; and the signal processing unit is configured toperform weighted addition for a second pixel signal output from thesecond pixel group by inter-frame processing, and to change a weight oneach frame in the weighted addition based on an inter-frame differentialof a first pixel signal.

According to another embodiment of the present invention, there isprovided an image processing method for processing a pixel signal outputfrom an imaging apparatus, the imaging apparatus including: an imagingdevice; and a signal processing unit, the imaging device including afirst pixel group and a second pixel group each including a plurality ofpixels each configured to output the pixel signal, the pixel signaloutput by the plurality of pixels of the second pixel group including asmaller amount of resolution information than an amount of resolutioninformation included in the pixel signal output by the plurality ofpixels of the first pixel group, the image processing method includingperforming weighted addition for a second pixel signal output from thesecond pixel group by inter-frame processing, and changing a weight oneach frame in the weighted addition based on an inter-frame differentialof a first pixel signal.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an imaging apparatus according to a firstembodiment of the present invention.

FIG. 2 is a block diagram of an imaging device according to the firstembodiment.

FIG. 3 is a circuit diagram of the imaging device and a columnamplifying unit according to the first embodiment.

FIG. 4A, FIG. 4B, FIG. 4C and FIG. 4D are diagrams for illustratingexamples of a color filter array using RGB.

FIG. 5A, FIG. 5B, FIG. 5C and FIG. 5D are diagrams for illustratingexamples of a color filter array using complementary colors.

FIG. 6 is a block diagram of a signal processing unit of the imagingapparatus according to the first embodiment.

FIG. 7A, FIG. 7B and FIG. 7C are diagrams for illustrating processing ofa control unit according to the first embodiment.

FIG. 8A and FIG. 8B are diagrams for illustrating inter-frame processingconducted by a color signal processing unit according to the firstembodiment.

FIG. 9A, FIG. 9B and FIG. 9C are a diagram and graphs for illustratingand showing operations of the control unit and the color signalprocessing unit according to the first embodiment.

FIG. 10 is a table for showing evaluation results of the imagingapparatus according to the first embodiment.

FIG. 11 is a block diagram of the signal processing unit of an imagingapparatus according to a second embodiment of the present invention.

FIG. 12A and FIG. 12B are graphs for showing an action of inter-frameprocessing according to the second embodiment.

FIG. 13 is a block diagram of the signal processing unit of an imagingapparatus according to a third embodiment of the present invention.

FIG. 14A and FIG. 14B are illustrations of an operation of adiscrimination processing unit according to the third embodiment.

FIG. 15 is a block diagram of the signal processing unit of an imagingapparatus according to a fourth embodiment of the present invention.

FIG. 16 is a block diagram of the signal processing unit of an imagingapparatus according to a fifth embodiment of the present invention.

FIG. 17 is a block diagram of the signal processing unit of an imagingapparatus according to a sixth embodiment of the present invention.

FIG. 18 is a block diagram of the signal processing unit of an imagingapparatus according to a seventh embodiment of the present invention.

FIG. 19 is a diagram for illustrating an example of a configuration ofan imaging system according to an eighth embodiment of the presentinvention.

DESCRIPTION OF THE EMBODIMENTS

An imaging apparatus according to each embodiment of the presentinvention is described with reference to the accompanying drawings. Inthe following description, like components are denoted by like referencesymbols.

First Embodiment

FIG. 1 is a block diagram of an imaging apparatus according to a firstembodiment of the present invention. The imaging apparatus includes animaging device 1 and a signal processing unit 2. The imaging device 1 isa so-called single-plate color sensor in which color filters arearranged on a CMOS image sensor or on a CCD image sensor. When a colorimage is formed with a single-plate color sensor, interpolation needs tobe conducted as described later. For example, an R pixel has noinformation (pixel value) of G or B. Therefore, based on pixel values ofG and B around the R pixel, pixel values of G and B in the R pixel aregenerated by interpolation processing. The imaging device 1 includes aplurality of pixels arranged in a matrix shape, for example, includes2,073,600 pixels in total of 1,920 pixels in a column direction and1,080 pixels in a row direction. The number of pixels of the imagingdevice 1 is not limited thereto, and may be a larger number of pixels ora smaller number of pixels. In addition, the imaging apparatus may notnecessarily include the imaging device 1, and it suffices that theimaging apparatus include the signal processing unit 2 configured toprocess a pixel signal (RAW data) received from the imaging device 1.

CFs according to this embodiment use an RGBW12 array illustrated inFIG. 1. In the RGBW12 array, a 4×4 pixel array is repeated, and a ratioof the numbers of pixels among the respective colors isR:G:B:W=1:2:1:12. In the RGBW12 array, the pixels of R, G, and B beingcolor pixels are each surrounded by eight W pixels, and the proportionof the W pixels accounts for ¾ of all the pixels. In other words, theRGBW12 array includes W pixels as a first pixel group, and includescolor pixels (RGB pixels) as a second pixel group. A total sum of thenumber of pixels of the first pixel group is three or more times larger(more than two times or larger) than a total sum of the number of pixelsof the second pixel group, and the second pixel group has a smalleramount of resolution information than that of the first pixel group. Theimaging device 1 can include not only effective pixels but also pixelsthat do not output an image, such as an optical black pixel and a dummypixel that does not include a photoelectric converter. However, theoptical black pixel or the dummy pixel is not included in the firstpixel group or the second pixel group. The W pixel has a wider spectralsensitivity characteristic and a higher sensitivity than those of theRGB pixel. For example, a wavelength half width of the spectralsensitivity characteristic of the W pixel is larger than that of the RGBpixel.

In the RGBW12 array, the W pixels are arranged around each of the RGBpixels, and hence a W pixel value in the position of the RGB pixel canbe interpolated with high accuracy. The W pixels account for ¾ of allthe pixels, and thus the sensitivity can be improved. This embodiment isparticularly effective for an imaging device in which the pixels forobtaining resolution information account for a half or more of all thepixels.

The signal processing unit 2 includes a pre-processing unit 203, aluminance signal processing unit 204, a control unit 205, a color signalprocessing unit 206, and a signal combining unit 207. A pixel signalreceived from the imaging device 1 is input to the pre-processing unit203. The pre-processing unit 203 executes various kinds of correctionincluding offset correction and gain correction for the pixel signal.When the pixel signal output from the imaging device 1 is an analogsignal, A/D conversion may be executed by the pre-processing unit 203.

The pre-processing unit 203 appropriately carries out correction such asoffset (OFFSET) correction and gain (GAIN) correction for an input pixelsignal Din to generate a corrected output Dout. This processing isexpressed typically by the following expression.Dout=(Din −OFFSET)·GAIN

This correction can be conducted in units of various circuits. Forexample, the correction may be conducted for each pixel. In addition,the correction may be conducted for each of circuits of a columnamplifier, an analog-to-digital conversion unit (ADC), and an outputamplifier. Through the correction, so-called fixed pattern noise isreduced, and an image with higher quality can be obtained. Thepre-processing unit 203 separates an image signal of W for resolutioninformation (luminance signal) and a pixel signal of RGB for colorinformation (color signal) to output the luminance signal to theluminance signal processing unit 204 and output the color signal to thecolor signal processing unit 206.

The luminance signal processing unit 204 can interpolate the signalreceived from the pre-processing unit 203 with high accuracy. That is,in the RGBW12 array, there are a large number of W pixels for obtainingresolution information, and hence it is possible to obtain informationhaving a higher spatial frequency, namely, a finer pitch, than the CFarray having a checkered pattern. Therefore, a pixel value of a part inwhich an RGB pixel exists, that is, a part in which no W pixel exists,is obtained from an average of pixel values of surrounding eight Wpixels, to thereby be able to obtain an image having a sufficiently highresolution. In another case, an edge may be detected based on edgeinformation and information such as a cyclic shape to interpolate the Wpixel in the position of the RGB pixel. In this case, it is possible toobtain an image having a higher resolution than in the case of using theaverage of the surrounding eight pixels. In the following, the W pixelgenerated by interpolation is represented as iW.

The color signal processing unit 206 conducts inter-frame processing forthe pixel signal of RGB being a color signal, to thereby reduce colornoise such as false colors. The inter-frame processing is conducted bysubjecting a plurality of frames different in time to weighted addition.The color signal processing unit 206 generates color information to beused for combining the luminance signal and the color signal. Thecontrol unit 205 determines a change (correlation) of the luminancesignal between frames, and changes the number of frames (weight) to beused for the inter-frame processing conducted by the color signalprocessing unit 206 based on a determination result. With thisoperation, it is possible to suppress the color noise such as a colorafterimage caused by an object exhibiting a large motion.

FIG. 2 is a block diagram of the imaging device 1 according to thisembodiment. The imaging device includes an imaging area 101, a verticalscanning circuit 102, a column amplifying unit 103, a horizontalscanning circuit 104, and an output unit 105. As described above, theimaging area 101 has pixels 100 arranged in a matrix shape, and includesthe first pixel group for a luminance signal and the second pixel groupfor a color signal. The vertical scanning circuit 102 supplies a controlsignal for controlling a transistor of the pixel 100 between an on state(conducting state) and an off state (non-conducting state). A verticalsignal line 106 is provided to each column of the pixels 100, and readssignals from the pixels 100 column by column. The horizontal scanningcircuit 104 includes a switch connected to an amplifier of each column,and supplies a control signal for controlling the switch between an onstate and an off state. The output unit 105 is formed of a bufferamplifier, a differential amplifier, or the like, and outputs the pixelsignal received from the column amplifying unit 103 to the signalprocessing unit 2 outside the imaging device 1. The output pixel signalis subjected to processing such as analog-to-digital conversion andcorrection of the input data by the signal processing unit 2.

The imaging device 1 may also be a so-called digital sensor having ananalog-to-digital conversion function. The pixel 100 includes CFs forcontrolling a spectral sensitivity characteristic, and in thisembodiment, CFs of RGBW12 are arranged.

FIG. 3 is a circuit diagram of the pixel 100 and the column amplifyingunit 103 of the imaging device 1 according to this embodiment. In thiscase, in order to facilitate description, a circuit corresponding to onecolumn within the column amplifying unit 103 and one pixel 100 areillustrated. The pixel 100 includes a photodiode PD, a stray diffusioncapacitance FD, a transfer transistor M1, a reset transistor M2, anamplifying transistor M3, and a selection transistor M4. The pixel 100may also be configured so that a plurality of photodiodes PD share thestray diffusion capacitance FD, the reset transistor M2, the amplifyingtransistor M3, and the selection transistor M4. The transistors M2 to M4are not limited to an N-channel MOS, and may also be formed of aP-channel MOS.

The photodiode PD is configured to photoelectrically convert appliedlight into an electron (charge). A signal TX is supplied to a gate ofthe transfer transistor M1, and when the signal TX is set to a highlevel, the transfer transistor M1 transfers the charge generated in thephotodiode PD to the stray diffusion capacitance FD. The stray diffusioncapacitance FD serves as a drain terminal of the transfer transistor M1,and can hold the charge transferred from the photodiode PD via thetransfer transistor M1. A signal RES is supplied to a gate of the resettransistor M2, and when the signal RES is set to a high level, the resettransistor M2 resets the voltage of the stray diffusion capacitance FDto a reset voltage VDD. When the transfer transistor M1 and the resettransistor M2 are simultaneously turned on, the electron of thephotodiode PD is reset. A gate of the amplifying transistor M3 isconnected to the stray diffusion capacitance FD.

A source of the amplifying transistor M3 is electrically connected to anode PDOUT of the vertical signal line 106 common to each column via theselection transistor M4 to form a source follower. A signal SEL isapplied to a gate of the selection transistor M4, and when the signalSEL is set to a high level, the vertical signal line 106 and theamplifying transistor M3 are electrically connected to each other. Withthis arrangement, a pixel signal is read from the selected pixel 100.

The signal TX, the signal RES, and the signal SEL to be supplied to thepixel 100 are output from the vertical scanning circuit 102. Thevertical scanning circuit 102 controls signal levels of those signals,to thereby scan the pixels 100 in units of rows. A current source 107supplies a current to the pixel 100 via the vertical signal line 106,and the vertical signal line 106 is connected to the column amplifyingunit 103 via a switch SW0 driven by the signal PL.

The column amplifying unit 103 includes a column amplifier 112, an inputcapacitance C0, feedback capacitances C1 and C2, switches SW1 to SW7,and capacitances CTN and CTS. The column amplifier 112 is formed of adifferential amplifier circuit including an inverted input node, anon-inverted input node, and an output node. The inverted input node ofthe column amplifier 112 is electrically connected to the verticalsignal line 106 via the switch SW0 and the input capacitance C0, and areference voltage VREF is applied to the non-inverted input node. Theinverted input node and the output node are connected to each other viathree feedback circuits that are connected in parallel. A first feedbackcircuit is formed of the switch SW1 and the feedback capacitance C1 thatare connected in series, a second feedback circuit is formed of theswitch SW2 and the feedback capacitance C2 that are connected in series,and a third feedback circuit is formed of the switch SW3. Anamplification factor of the column amplifier 112 can be changed byappropriately controlling the on state and the off state of the switchesSW1 to SW3. That is, when only the switch SW1 is turned on, theamplification factor becomes C0/C1, and when only the switch SW2 isturned on, the amplification factor becomes C0/C2. Further, when theswitches SW1 and SW2 are turned on, the amplification factor becomesC0/(C1+C2), and when only the switch SW3 is turned on, the columnamplifier 112 operates as a voltage follower. The switches SW1 to SW3are controlled by signals φC1, φC2, and φC, respectively.

The output node of the column amplifier 112 is connected to thecapacitance CTN via the switch SW4 controlled by a signal φCTN. In thesame manner, the output node of the column amplifier 112 is connected tothe capacitance CTS via the switch SW5 controlled by a signal φCTS. Whenthe stray diffusion capacitance FD is reset, the switch SW4 is turnedon, the switch SW5 is turned off, and a pixel signal (N signal) at atime of the resetting is sampled and held the capacitance CTN. After thephotoelectrically-converted charge is transferred to the stray diffusioncapacitance FD, the switch SW4 is turned off, the switch SW5 is turnedon, and a pixel signal (S signal) based on thephotoelectrically-converted charge is sampled and held by thecapacitance CTS.

The capacitance CTN is connected to a first input node of the outputunit 105 via the switch SW6, and the capacitance CTS is connected to asecond input node of the output unit 105 via the switch SW7. Thehorizontal scanning circuit 104 sets a signal φHn of each column to ahigh level in order, to thereby conduct horizontal scanning. That is,when the signal φHn is set to a high level, the switch SW6 outputs the Nsignal held by the capacitance CTN to the first input node of the outputunit 105, and the switch SW7 outputs the S signal held by thecapacitance CTS to the second input node of the output unit 105.

The output unit 105 is formed of a differential amplifier circuit, andamplifies and outputs a differential between the input S signal and Nsignal, to thereby output a pixel signal from which a noise component atthe time of the resetting has been removed. The output unit 105 may beconfigured to subject the N signal and the S signal to theanalog-to-digital conversion and then to correlated double sampling.

As described above, an optical signal input to the imaging device 1 isread as an electric signal. Further, two-dimensional information of aspectral intensity corresponding to the CF array of RGBW12 is obtained.This embodiment is not limited to the CF array of RGBW12, and can beapplied to various CF arrays. Examples of the CF array to which thisembodiment can be applied are described below.

FIG. 4A to FIG. 4D are illustrations of examples of a CF array using RGBas color pixels. FIG. 4A is an illustration of CFs of a Bayer array, anda ratio of the numbers of CFs is R:G:B=1:2:1. In this case, a largernumber of G pixels (first pixels) than the number of RB pixels (secondpixels) are arranged because a human visual characteristic has a highersensitivity to a wavelength of green than wavelengths of red and blue,and because a sense of resolution of an image depends on the wavelengthof green more strongly than red and blue.

FIG. 4B is an illustration of the CF array of RGBW12. As describedabove, in this array, the respective CFs are arranged at the ratio ofR:G:B:W=1:2:1:12 in the 4×4 pixel array. W pixels (first pixels) arearranged adjacent to each of RGB pixels (second pixels) being colorpixels in a vertical direction, a horizontal direction, and an obliquedirection in a plan view. That is, the RGB pixels are each surrounded byeight W pixels. The proportion of the W pixels accounts for ¾ of all thepixels. The RGB pixels being color pixels are each surrounded by the Wpixels.

FIG. 4C is an illustration of a CF array of RGBW8. In the 4×4 pixelarray, respective CFs are arrayed at the ratio of R:G:B:W=2:4:2:8. The Wpixels (first pixels) are arranged in a checkered pattern, and an RGBpixel (second pixel) is arranged among the W pixels. The proportion ofthe W pixels is ½ of all the pixels. The W pixels are arranged in acheckered pattern in the same manner as the G pixels within the Bayerarray, and hence a method of interpolating the G pixel of the Bayerarray can be used as it is. The arrangement of the W pixels allows animprovement in the sensitivity.

FIG. 4D is an illustration of a CF array of RGBG12. In this array, the Wpixels of RGBW12 are replaced by G pixels (first pixels), and in the 4×4pixel array, CFs of the respective colors are arranged at the ratio ofR:G:B=2:12:2. RB pixels (second pixels) are each surrounded by the Gpixels, and the proportion of the G pixels accounts for ¾ of all thepixels. The RB pixels are each surrounded by the G pixels, and hence theaccuracy improves in the interpolation of the G value of the colorpixel. The proportion of the G pixels, which have a higher sensitivitythan the RB pixels, is large, and hence the sensitivity can be improved.

FIG. 5A to FIG. 5D are illustrations of examples of a CF array usingcyan (C), magenta (M), and yellow (Y) which are complementary colors, ascolor pixels. FIG. 5A is an illustration of the Bayer array, and theratio of the CFs of the respective colors is C:M:Y=1:1:2. In this case,a large number of Y pixels (first pixels) are arranged because the Ypixel has a high sensitivity in the same manner as the G pixel.

FIG. 5B is an illustration of a CF array of CMYW12. In the 4×4 pixelarray, the CFs of the respective colors are arrayed at the ratio ofC:M:Y:W=1:1:2:12. CMY pixels (second pixels) being color pixels are eachsurrounded by W pixels (first pixels). The proportion of the W pixelsaccounts for ¾ of all the pixels. The CMY pixels are each surrounded bythe W pixels, and hence the accuracy can be improved in theinterpolation of a W pixel value in the position of the CMY pixel. Thearrangement of the W pixels allows an improvement in the sensitivity.

FIG. 5C is an illustration of a CF array of CMYW8. In the 4×4 pixelarray, the CFs of the respective colors are arrayed at the ratio ofC:M:Y:W=2:2:4:8. The W pixels (first pixels) are arranged in a checkeredpattern, and the CMY pixels (second pixels) are each surrounded by the Wpixels. The proportion of the W pixels is ½ of all the pixels. The Wpixels are arranged in a checkered pattern in the same manner as the Gpixels within the Bayer array, and hence a method of interpolating the Gpixel of the Bayer array can be used as it is. The arrangement of the Wpixels allows an improvement in the sensitivity.

FIG. 5D is an illustration of a CF array of CMYY12. The W pixels ofCMYW12 are replaced by Y pixels (first pixels), and in the 4×4 pixelarray, the respective CFs are arranged at the ratio of C:M:Y=2:2:12. TheC pixel and the M pixel (second pixels) are each surrounded by the Ypixels, and the proportion of the arranged Y pixels accounts for ¾ ofall the pixels. The C pixel and the M pixel are each surrounded by the Ypixels, and hence the accuracy can be improved in the interpolation ofthe pixel value of Y in the position of each of the C pixel and the Mpixel. The proportion of the Y pixels, which have a relatively highersensitivity than the C pixel and the M pixel, is large, and hence thesensitivity improves.

As described above, various CF arrays can be employed in thisembodiment, but in order to generate an image having a high resolution,it is preferred to arrange a larger number of pixels (first pixels) thatcontribute to the resolution to a larger extent. It is desired that thefirst pixel group include a larger amount of resolution information thanthat of the second pixel group, and that the second pixel group includeat least two kinds of pixels different in spectral sensitivity. It isdesired that the first pixel group have a higher degree of contributionto the luminance than the second pixel group.

In the Bayer array, the G pixels that contribute to the resolution arearranged in a checkered pattern, which is liable to cause aninterpolation error. The inventors of the present invention found thatthe interpolation error can be minimized through use of a CF array thatyields a higher resolution than the checkered pattern. Therefore, theeffects of the present invention are particularly noticeable through useof the CF arrays exemplified in RGBW12 of FIG. 4B, RGBG12 of FIG. 4D,CMYW12 of FIG. 5B, and CMYY12 of FIG. 5D.

FIG. 6 is a block diagram of the signal processing unit 2 of the imagingapparatus according to this embodiment. The signal processing unit 2includes the luminance signal processing unit 204, the control unit 205,the color signal processing unit 206, and the signal combining unit 207,and is configured to conduct demosaicing processing for a pixel signal 3a received from the imaging device 1 to generate an image signal 3 gincluding information of RGB for each pixel. The signal processing unit2 can be configured by hardware such as an image processing processor,but the same configuration can be implemented through use of ageneral-purpose processor or software on a computer.

The pixel signal 3 b, which includes a CF array of RGBW12 and isexpressed by digital data, is input to the luminance signal processingunit 204. In FIG. 6, 4×4 pixels serving as one unit of repetition of theCF array are illustrated, but in the actual pixel signal 3 a, the arrayof the 4×4 pixels is repeated. The input pixel signal 3 a is separatedinto a pixel signal 3 b of W and a pixel signal 3 e of RGB by thepre-processing unit 203 (not shown).

The luminance signal processing unit 204 includes an interpolationprocessing unit 211, and the interpolation processing unit 211 isconfigured to generate a pixel value of a part in which a pixel value ofW does not exist within the pixel signal 3 b of W by interpolation.There is no pixel value of W existing in positions from which RGB pixelshas been separated within the pixel signal 3 b of W, and in FIG. 6,those positions are each represented by “?”. The interpolationprocessing unit 211 interpolates the pixel value in the position of “?”based on the surrounding pixel values of W to generate pixel values ofiWr, iWg, and iWb by interpolation. For example, there is no W pixelexisting at coordinates (3,3) within the pixel signal 3 b, and hence thepixel value of iWb (3,3) at the coordinates (3,3) is obtained from anaverage value of the surrounding eight W pixel values as expressed bythe following expression.

${iWb}_{({3,3})} = \frac{W_{({2,2})} + W_{({3,2})} + W_{({4,2})} + W_{({2,3})} + W_{({4,3})} + W_{({2,4})} + W_{({3,4})} + W_{({4,4})}}{8}$

In FIG. 6, the 4×4 pixel array is illustrated, but in actuality, thepixel array is repeated, and each of an R pixel at coordinates (1,1), aG pixel at coordinates (3,1), and a G pixel at coordinates (1,3) issurrounded by eight W pixels. Therefore, the pixel values of iWr and iWgcan also be generated by interpolation through use of the surroundingeight pixel values of W in the same manner.

Examples of an interpolation processing method that can be appropriatelyused include not only the above-mentioned method but also a bilinearmethod and a bicubic method.

The control unit 205 includes a spatial average processing unit 212 anda discrimination processing unit 213. The spatial average processingunit 212 is configured to calculate an average value W_(ave) of pixelsfor each predetermined block within the pixel signal 3 b of W. Thediscrimination processing unit 213 is configured to compare aninter-frame differential of the average value W_(ave) with a thresholdvalue, and to output a discrimination signal J based on a comparisonresult. The color signal processing unit 206 includes an inter-frameprocessing unit 214, and the inter-frame processing unit 214 isconfigured to conduct inter-frame processing for the pixel signal 3 e ofRGB to reduce the color noise or false colors. The signal combining unit207 is configured to calculate the color ratio information of therespective RGB pixels subjected to the inter-frame processing and the Wpixels, and to generate an image signal 3 g expressed by pixel values ofRGB based on the color ratio information.

FIG. 7A and FIG. 7B are illustrations of examples of the pixel area tobe used for spatial average processing conducted by the control unit205. One block of the RGBW12 pixel array is formed of 4×4 pixels, andhence the average value of the pixel values of W included in one blockcan be calculated. That is, in FIG. 7A, the average value W_(ave) can becalculated for each block by arithmetically operating spatial average ofthe W values of 12 pixels at (x,y)=(2,1), (4,1), (1,2), (2,2), (3,2),(4,2), (2,3), (4,3), (1,4), (2,4), (3,4), and (4,4). The average valueW_(ave) calculated in this manner is used in discrimination of theinter-frame processing for the RGB pixels at (x,y), (1,1), (3,1), (1,3),and (3,3) included in the same block.

FIG. 7B is an illustration of another example of the pixel area for thespatial average processing. In the RGBW12 pixel array, each of the RGBpixels is surrounded by the W pixels. For example, the B pixel at(x,y)=(3,3) is surrounded by eight W pixels of (x,y)=(2,2), (3,2),(4,2), (2,3), (4,3), (2,4), (3,4), and (4,4). Therefore, the averagevalue W_(ave) of the eight W pixels may be used for the discriminationof the inter-frame processing for the B pixel at (3,3). In the samemanner, the average value W_(ave) of the eight W pixels around the Gpixel at (5,3) can be used for the discrimination of the inter-frameprocessing for the G pixel at (5,3). In any one of the examples of FIG.7A and FIG. 7B, a plurality of pixel values of W around the RGB pixelsto be subjected to the inter-frame processing are used to calculate theaverage value W_(ave) for each pixel area. The spatial averageprocessing may be conducted through use of not only a simple average butalso a weighted average, a smoothing filter, or the like.

FIG. 7C is an illustration of details of processing conducted by thediscrimination processing unit 213. An average value W_(ave) (N) of theW pixels in the N-th frame and an average value W_(ave) (N−1) of the Wpixels in the (N−1)th frame are stored in frame memory. Thediscrimination processing unit 213 compares an absolute value of adifferential between the average value (N) and the average value W_(ave)(N−1) with a predefined threshold value Vth, and outputs thediscrimination signal (signal value) J based on the comparison resultfor each pixel area. When the absolute value of the differential islarger than the threshold value Vth, the discrimination signal J is “1”,and when the absolute value of the difference is equal to or smallerthan the threshold value Vth, the discrimination signal J is “0”(Mathematical Expression 3). The discrimination signal J indicateswhether or not there has been a change in the luminance within the pixelarea included in the average value W_(ave), that is, a temporal changeof the object.|W _(ave)(N)−W _(ave)(N−1)|>Vth→J=1|W _(ave)(N)−W _(ave)(N−1)|≦Vth→J=0

When an object moves, it is a rare case that there is a change only incolors, and the luminance usually changes. For this reason, a temporalchange of the object can be detected with higher accuracy by detectingthe differential of the W pixel value exhibiting a high sensitivityinstead of detecting the differential of the RGB pixel value. In theRGBW12 array according to this embodiment, the number of W pixels islarger than the number of RGB pixels. Therefore, through use of thespatial average value of the W pixel, it is possible to further reducethe noise of the W pixel, to thereby be able to avoid the influence ofthe noise in the discrimination conducted between frames.

FIG. 8A and FIG. 8B are illustrations of the inter-frame processingconducted by the inter-frame processing unit 214. In this case, theinter-frame processing for the B pixel is illustrated, but the sameprocessing is also conducted for the pixels of R and G. The inter-frameprocessing unit 214 uses an IIR filter (recursive filter) to conductweighted addition for each of the pixel value in the current frame andthe pixel value in another frame different in time. In the inter-frameprocessing of FIG. 8A, the weighted addition is conducted through use ofany one of factors m and n (n>m) based on the discrimination signal J.

When the discrimination signal J is “0”, that is, when the temporalchange in the luminance within the pixel area to be subjected to theinter-frame processing is small, the inter-frame processing unit 214adds a value obtained by multiplying the pixel value accumulated in theframe memory by the factor (n−1)/n and a value obtained by multiplyingthe current pixel value of B by the factor 1/n to obtain an inter-frameprocessed pixel value of n_B. When the discrimination signal J is “1”,that is, when the temporal change in the luminance within the pixel areais large, the inter-frame processing unit 214 adds a value obtained bymultiplying the pixel value accumulated in the frame memory by thefactor (m−1)/m and a value obtained by multiplying the current pixelvalue of B by the factor 1/m to obtain the inter-frame processed pixelvalue of n_B. The inter-frame processed pixel value of n_B isaccumulated in the frame memory, and is subjected to the weightedaddition in the next inter-frame processing. That is, the pixel value inthe past frame is fed back to the pixel value in the next frame, and theaddition averaging is conducted. In this case, it is assumed that n>m,and hence in a pixel area exhibiting a large temporal change in theluminance, the weight on the past frame is relatively smaller than theweight on the current frame. Meanwhile, in a pixel area exhibiting asmall temporal change in the luminance, the weight on the past frame isrelatively larger than the weight on the current frame. Note that, inthe following description, “changing the weight” refers to increasing orreducing the weight on a frame, or also refers to reducing the weight tozero. The reducing the weight on the past frame or reducing the weightto zero may also be referred to as “reducing the number of processedframes”.

As another example of the inter-frame processing, processing illustratedin FIG. 8B may be employed. When the discrimination signal J is “0”, theinter-frame processing unit 214 adds the value obtained by multiplyingthe pixel value accumulated in the frame memory by the factor (n−1)/nand the value obtained by multiplying the pixel value of B in thecurrent frame by the factor 1/n to obtain the inter-frame processedpixel value of n_B. When the discrimination signal J is “1”, theinter-frame processing unit 214 outputs the pixel value of B in thecurrent frame as it is, and further stores the pixel value of B in thecurrent frame into the frame memory. This processing is the sameprocessing conducted in the case where m=1 in FIG. 8A. In this manner,in the pixel area exhibiting a large temporal change in the luminance,the weight on the current frame is increased in the inter-frameprocessing, to thereby be able to reduce color blur and an afterimagethat are caused by the addition of the pixel value in the past frame.

In the above description, the discrimination signal J is a one-bitsignal representing “0” and “1”, but may be a signal having two bits ormore. That is, a magnitude (differential) of the motion of an object maybe expressed by the discrimination signal J having a plurality of bits,and the weight on a color signal for the inter-frame processing or thenumber of frames for the inter-frame processing may be changed(increased or reduced) based on the magnitude of the discriminationsignal J. In this case, the prevention of a colored afterimage and thereduction in color noise can be maintained at an optimum balancedepending on the object.

The signal combining unit 207 generates the image signal 3 g includingRGB information for each pixel based on the luminance signal subjectedto the interpolation processing and the color signal or pixel signalsubjected to the inter-frame processing. The processing of the signalcombining unit 207 is described below in detail. In FIG. 6, the signalcombining unit 207 arithmetically operates the color ratio (colorinformation) in each pixel based on a pixel signal 3 c subjected to theinterpolation and the inter-frame processed pixel values of n_R, n_G,and n_B. On the assumption that the color ratio is constant within the4×4 pixel area, the color information can be calculated in the followingmanner. That is, the color information of R is expressed by n_R/iWr atthe coordinates (1,1), and the color information of B is expressed byn_B/iWb at the coordinates (3,3). Further, the color information of G isexpressed by an average value between n_G/iWg at the coordinates (3,1)and n_G/iWg at the coordinates (1,3).

The signal combining unit 207 generates the image signal 3 g includinginformation of the respective colors of RGB for each pixel on theassumption that the ratio among the respective colors is constant withinthe 4×4 area. That is, the signal combining unit 207 uses a pixel signal3 c of W and iW generated by the luminance signal processing unit 204and the color information to generate the image signal 3 g. When thepixel of the pixel signal 3 c is W, the pixel value of RGB is obtainedby the following expression.

${RGB} = \begin{bmatrix}{\frac{n\_ R}{iWr}W} & {\frac{n\_ G}{iWg}W} & {\frac{n\_ B}{iWb}W}\end{bmatrix}$

Further, when the pixel of the pixel signal 3 c is iW, the pixel valueof RGB is obtained by the following expression.

${RGB} = \begin{bmatrix}{\frac{n\_ R}{iWr}{iW}} & {\frac{n\_ G}{iWg}{iW}} & {\frac{n\_ B}{iWb}{iW}}\end{bmatrix}$

The color information is standardized by the W pixel value or the iWpixel value. That is, the color information expressed by n_R/iWr,n_G/iWg, and n_B/iWb does not include the luminance informationnecessary for the resolution, and includes only color information (hueinformation). Therefore, the pixel value of RGB can be obtained bymultiplying the pixel values of W and iW being luminance information bythe color information. The pixel values of W and iW being luminanceinformation are not subjected to the inter-frame processing, or are notsubjected to low pass filter processing in a sense of time or frequency.This enables generation of an image signal superior in response speedwith respect to the motion of the object. The color signal is subjectedto the inter-frame processing, and the noise in the pixel values of n_R,n_G, and n_B is reduced, to thereby be able to generate a satisfactoryimage signal exhibiting few false colors.

In the human visual characteristic, the respective resolution powers(recognition capabilities) of the resolution (luminance) and the color(hue) are different from each other. The capability of sensing colors isnot “spatially” or “temporally” high, and the resolution power of thecolor hardly becomes a problem compared to that of the luminance. Thatis, as long as the response to the luminance is fast, the RGB image islikely to be recognized as being satisfactory. Meanwhile, when there isa change in the hue that is not included in the object, that is, thefalse color or the noise component, a sense of discomfort in terms of avisual characteristic becomes large, and the image quality deteriorates.According to this embodiment, the color signal is subjected to theinter-frame processing, and hence an image having little noise can begenerated. Further, when the motion of the object is large, afterimagesof colors can be reduced by lowering the weight on the past frame in theinter-frame processing.

FIG. 9A to FIG. 9C are a diagram and graphs for illustrating and showingoperations of the control unit 205 and the color signal processing unit206 according to this embodiment, and an imaging result of such anobject that a blue sphere moves in front of a gray background isillustrated. FIG. 9A is an illustration of a map indicating two frames((N−1)th frame and N-th frame) different in time and the discriminationsignal J on a two-dimensional image. When the blue sphere moves and theabsolute value of the differential between the spatial average valuesW_(ave) (N) and W_(ave) (N−1) in the two frames exceeds the thresholdvalue Vth, the discrimination signal J becomes “1”. Therefore, in thearea where the discrimination signal J is “1” in FIG. 9A, theinter-frame processing for the color signal is not conducted, or theweight on the past frame becomes small. In an area exhibiting a smallchange in the luminance, that is, the area where the absolute value ofthe differential between the spatial average values is equal to orsmaller than the threshold value Vth, the discrimination signal J is“0”, and the color signal is subjected to the inter-frame processing toreduce the color noise.

FIG. 9B is a graph for showing the spatial average value W_(ave) and thepixel value (output value) of B in each frame of a rectangular area 900of FIG. 9A. In the (N−3)th frame to the (N−1)th frame, the blue sphereis positioned over the area 900, and hence the pixel value of B withinthe area 900 is a value shown in the graph. The spatial average valueW_(ave) calculated from the pixel value of W is at a higher level thanthe pixel value of B. When the blue sphere further moves and the area900 changes from blue to gray during a period from the (N−1)th frame tothe N-th frame, the pixel value of B and the spatial average valueW_(ave) change. When the absolute value of the differential between thespatial average values W_(ave) (N) and W_(ave) (N−1) exceeds thethreshold value Vth, the discrimination signal J becomes “1”, and thecolor signal in the N-th frame is not subjected to the frame inter-frameprocessing.

FIG. 9C is a graph for showing a result of comparing the discriminationprocessing between the frames in the examples of FIG. 9A and FIG. 9B.The graph on the left of FIG. 9C is a graph for showing the respectivepixel values of W and B in the N-th frame obtained when thediscrimination processing between the frames is not conducted. In thiscase, the pixel value of B subjected to the inter-frame processing isthe pixel value of n_B substantially the same as the average value offrom the (N−3)th frame to the N-th frame. Meanwhile, the pixel value ofW is the value of the N-th frame, and hence the color ratio of the pixelvalues of W and n_B greatly deviates from an original color ratio forgray, which causes the gray area to be colored blue. Therefore, in thedisplayed moving image, a belt-like afterimage that follows the movingblue sphere is visually recognized.

The graph on the right of FIG. 9C is a graph for showing the respectivepixel values of W and B in the N-th frame obtained when thediscrimination processing between the frames is conducted according tothis embodiment. In this case, in the area where the absolute value ofthe differential between the spatial average values W_(ave) in therespective frames is large, the inter-frame processing for the colorsignal is not conducted. Therefore, the pixel value of B is the value ofthe N-th frame, and the color ratio of the pixel values of W and B isthe original color ratio for gray. That is, in the area where the motionof the object is large, the color ratio is not affected by the pastframe, and hence the above-mentioned afterimage of blue does not occur.

In the part exhibiting a large motion, the noise is less liable tobecome conspicuous, and hence the deterioration in image quality can besuppressed to a minimum even by inhibiting the inter-frame processingfrom being conducted for the color signal in the part exhibiting a largetemporal change of the W pixel. In the part where the temporal change ofthe W pixel is small, the color noise can be reduced by conducting theinter-frame processing for the color signal. Therefore, both thereduction in color noise and the reduction in color afterimages can beachieved for an entire image, and a high-quality image can be obtained.

FIG. 10 is a table for showing evaluation results of the imagingapparatus according to this embodiment. As evaluation items for animage, noise and an afterimage were used. Interference due to a falsecolor in a moving image was represented by “A”, “B”, and “C” in orderfrom an excellent evaluation. The evaluation was conducted with aluminance, a number n of frames, and presence or absence of framediscrimination processing being changed as the evaluation conditions. Inthis case, the number n of frames represents “n” within the factors 1/nand (n−1)/n used in the inter-frame processing. The frame discriminationprocessing represents the discrimination based on the spatial averagevalue W_(ave) of the W pixels.

As Condition No 1, an ambient luminance was set to 0.1 [lx], the numberof frames was set as n=1, and discrimination processing was notconducted. A photographed image did not have a color afterimage whichappears as a blurred color, but cannot be said to have had satisfactoryimage quality due to color noise conspicuous enough to be annoying.Therefore, the evaluation of the noise was “C”, and the evaluation ofthe afterimage was “A”.

As Condition No 2, the ambient luminance was set to 0.1 [lx], the numberof frames was set as n=4, and the discrimination processing was notconducted. In the photographed image, the color afterimage was somewhatconspicuous, but was at an acceptable level. The color noise wasconspicuous, but was at the tolerable level. Therefore, the evaluationof the noise was “B”, and the evaluation of the afterimage was “A”.

As Condition No 3, the ambient luminance was set to 0.05 [lx], thenumber of frames was set as n=4, and the discrimination processing wasnot conducted. In the photographed image, the color afterimage was at anacceptable level, but the color noise was at an annoying level, andsatisfactory image quality failed to be obtained. Therefore, theevaluation of the noise was “C”, and the evaluation of the afterimagewas “A”.

As Condition No 4, the ambient luminance was set to 0.05 [lx], thenumber of frames was set as n=8, and the discrimination processing wasnot conducted. In the photographed image, the color noise was at anacceptable level. However, because the number of frames to be subjectedto averaging was increased, the color afterimage was conspicuous, andthe image quality was annoying. Therefore, the evaluation of the noisewas “A”, and the evaluation of the afterimage was “C”.

As Condition No 5, the ambient luminance 0.05 [lx], the number of frameswas set as n=8, and the discrimination processing was set conducted. Inthe photographed image, both the color noise and the color afterimagewere both at an acceptable level. Therefore, the evaluations of thenoise and the afterimage were both “A”.

As described above, according to this embodiment, instead of the Bayerarray of RGB, an imaging apparatus having a high sensitivity can berealized through use of CFs having the W pixel. Further, the humanvisual characteristic is not so sensitive to the temporal change in thecolor, and hence the color noise can be reduced by subjecting the colorsignal to the inter-frame processing and conducting noise reduction. Thepart exhibiting a large temporal change is discriminated to skip thenoise reduction, to thereby be able to reduce the color noise and thecolor afterimages while maintaining the resolution of the moving image.The frame discrimination processing is conducted based on the N pixelhaving a high resolution, and hence the accuracy in the discriminationcan also be increased.

In general, in order to achieve the high sensitivity, when theproportion of N pixels is increased in a given small area, theproportion of RGB pixels decreases, and the color noise increases. Thisis described as follows. The noise generated in each pixel is a sum ofphoton shot noise and readout noise, both of which have a characteristicbased on a Poisson distribution. Thus, a standard deviation in the smallarea is proportional to the square root of a number N of pixels.Meanwhile, an average value in a given small area is proportional to thenumber N of pixels. Therefore, an S/N ratio is proportional to thesquare root of the number N of pixels, and the S/N ratio becomes higheras the number of pixels becomes larger, that is, an image having smallnoise can be obtained. With the RGB pixels including a small number ofpixels, the color noise is liable to increase, but according to thisembodiment, the inter-frame processing enables the color noise to bereduced effectively. The color afterimages can also be reduced bychanging the weight for the inter-frame processing based on the objectexhibiting a large motion.

Note that, the inter-frame processing is not limited to the IIR filter,and a non-recursive filter (FIR) may be used, or an inter-frame movingaverage or an inter-frame median filter may be used. In addition, thenumber of frames for the inter-frame processing is not limited to afixed value such as 1, 4, and 8, and an adaptive filter configured toappropriately change the value of n depending on an environment(luminance, contrast, moving speed of an object, or the like) of anobject may be used. The frames to be used for the inter-frame processingare not limited to adjacent frames, and may be every plurality of framessuch as every two frames or every three frames. In addition, aninclination of the pixel values of three or more points among three ormore frames may be used to calculate a variation amount (differential orcorrelation value) between frames.

In this embodiment, an example in which the luminance signal processingunit 204 and the color signal processing unit 206 are provided outsidethe imaging device 1 is described. As another example, the imagingdevice 1 may include the luminance signal processing unit 204 and thecolor signal processing unit 206. The imaging device 1 may be alaminated sensor obtained by laminating a first semiconductor substratein which an imaging area 101 is formed and a second semiconductorsubstrate in which the luminance signal processing unit 204 and thecolor signal processing unit 206 are each formed on one another.

Second Embodiment

FIG. 11 is a block diagram of the signal processing unit 2 of an imagingapparatus according to a second embodiment of the present invention. Inthe first embodiment, the discrimination of the inter-frame processingis conducted based on the average value W_(ave) of the pixel values ofW, but in this embodiment, the discrimination of the inter-frameprocessing is conducted based on the color ratio information. Theimaging apparatus according to the second embodiment is described belowmainly in terms of points different from those of the first embodiment.

The control unit 205 includes a color ratio processing unit 222 and thediscrimination processing unit 213, and the pixel signal 3 c of Wsubjected to the interpolation and the pixel signal 3 e of RGB are inputto the color ratio processing unit 222.

The color ratio processing unit 222 is configured to calculate ratios ofthe pixel values of iWr, iWg, and iWb subjected to the interpolation andthe pixel values of R, G, and B, respectively, and to output the colorratio information. The color ratio is calculated in accordance withexpressions of R/iWr, G/iWg, and B/iWb.

The discrimination processing unit 213 is configured to compare theabsolute value of an inter-frame differential of the color ratioinformation with the threshold value Vth, and to output thediscrimination signal J based on the comparison result. For example, thediscrimination signal J based on a color ratio of B/iWb is obtained bythe following expressions.

${{{{\frac{B(N)}{{iWb}(N)} - \frac{B\left( {N - 1} \right)}{{iWb}\left( {N - 1} \right)}}} > {Vth}}->J} = 1$${{{{\frac{B(N)}{{iWb}(N)} - \frac{B\left( {N - 1} \right)}{{iWb}\left( {N - 1} \right)}}} \leq {Vth}}->J} = 0$

In the above-mentioned expressions, B(N) and iWb(N) represent the pixelvalue of B and the pixel value of iWb the N-th frame, and B(N−1) andiWb(N−1) represent the pixel value of B and the pixel value of iWb inthe (N−1)th frame. The discrimination signals J for the pixel values ofR and G can also be obtained in the same manner. In this case, thediscrimination signal J can be obtained for each of R, G, and B tochange the weight for the inter-frame processing. The discriminationsignal J may be determined based on the color ratio that causes theabsolute value of the differential to become maximum, and moreover, thediscrimination signal J may be determined through use of an averagevalue of the color ratios. It is desired that the threshold value Vth bea value equal to or smaller than 1 and be set to an optimum value basedon the imaging results of various objects. The threshold value Vth maybe changed dynamically based on photographing conditions such as anilluminance of the object and a color temperature of illumination.

FIG. 12A and FIG. 12B are graphs for showing the operations of thecontrol unit 205 and the color signal processing unit 206 according tothis embodiment. FIG. 12A is a graph for showing the respective pixelvalues of iWb and B in each frame, and FIG. 12B is a graph for showingthe color ratio information in each frame. In the same manner as FIG. 9Ato FIG. 9C, FIG. 12A and FIG. 12B are graphs for showing an imagingresult of such an object that a blue sphere moves in front of a graybackground, and for showing changes in the respective values within thearea 900 from the (N−3)th frame to the N-th frame. In the (N−3)th frameto the (N−1)th frame, the change in the color ratio information B/iWb issmall, and hence the absolute value of the inter-frame differential ofthe color ratio information is equal to or smaller than the thresholdvalue Vth. Therefore, the discrimination signal J is “0”, and theinter-frame processing unit 214 conducts the inter-frame processing forthe color signal, and conducts the noise reduction for the color signal.When the area 900 changes from blue to gray during the period from the(N−1)th frame to the N-th frame, the color ratio information of thepixel value of B and the pixel value of iWb changes. When the absolutevalue of the inter-frame differential of the color ratio informationB/iWb exceeds the threshold value Vth, the discrimination signal Jbecomes “1”, and the inter-frame processing for the color signal is notconducted. Therefore, the color afterimage that can be caused byreferring to the past frame can be suppressed.

In this embodiment, the weight for the inter-frame processing is changedbased on the color ratio, and hence an arithmetic operation processingamount can increase compared to the first embodiment in which the weightis changed based on the pixel value of W, but the color afterimage canbe suppressed effectively. A major cause of an occurrence of a colorafterimage due to the inter-frame processing (noise reduction) of thecolor signal processing is a drastic change in the color ratio.Therefore, it is possible to effectively suppress the color afterimageby discriminating whether or not to conduct the inter-frame processingfor the color signal based on the change in the color ratio(differential). In the RGBW12 array, the RGB pixels are each surroundedby the W pixels, and hence the pixel value of W in the position of theRGB pixel can be interpolated with high accuracy. Therefore, the colorratio can be calculated with high accuracy, and it is possible toimprove the accuracy in the discrimination of the inter-frameprocessing.

As described above, according to this embodiment, the weight on eachframe in the inter-frame processing for the color signal is changedthrough use of the color ratio. The color ratio is used for thediscrimination, to thereby be able to prevent the inter-frame processingfrom being conducted so as to deviate from the original color ratio, andto prevent an occurrence of a colored afterimage with high accuracy.That is, it is possible to conduct the discrimination of the inter-frameprocessing with higher accuracy than in the first embodiment, and thecolor afterimages can be further reduced.

In the discrimination of the inter-frame processing, a color differencebetween the pixel value of iW and the pixel value of RGB may be usedinstead of the color ratio. That is, when the color difference betweenthe frames is equal to or smaller than the threshold value Vth, theinter-frame processing for the color signal is executed, and when thecolor difference between the frames exceeds the threshold value Vth, theinter-frame processing for the color signal may be inhibited from beingexecuted.

Third Embodiment

FIG. 13 is a block diagram of the signal processing unit 2 of an imagingapparatus according to a third embodiment of the present invention. Thisembodiment is different from the first embodiment in that thediscrimination of the inter-frame processing is conducted based on amotion vector using the W pixels in a plurality of frames. The imagingapparatus according to this embodiment is described below mainly interms of points different from those of the first embodiment.

A discrimination processing unit 231 is configured to calculate a motionvector by a block matching method, and to determine the determinationsignal J based on a comparison result between the absolute value of themotion vector and the threshold value Vth.

FIG. 14A and FIG. 14B are illustrations of an operation of thediscrimination processing unit 231 according to this embodiment, and abasic frame and a reference frame for use in the block matching methodare illustrated. In FIG. 14A, the discrimination processing unit 231sets a window area A within the basic frame, and searches the referenceframe for a pattern similar to a pattern within the window area A. Asthe reference frame, the N-th frame subsequent to the (N−1)th frame isused.

As illustrated in FIG. 14B, normally, a predetermined range C with aposition B exhibiting a moving amount of zero being used as a referenceis set as a search range. A similarity between patterns can bedetermined by using a sum of squared difference (SSD) expressed by thefollowing expression as an evaluation value.

${SSD} = {\sum\limits_{x,{y \in W}}\;\left( {{f\left( {{x + u},{y + v},{t + {\Delta\; t}}} \right)} - {f\left( {x,y,t} \right)}} \right)^{2}}$

In this expression, f(x,y,t) represents a space-time distribution of animage, and x,yεW means coordinate values of a pixel included in a windowarea of the basic frame.

The discrimination processing unit 231 changes (u,v) within the searchrange to search for a combination of (u,v) exhibiting the minimumevaluation value, and sets this (u,v) as the motion vector between theframes. The discrimination processing unit 213 sequentially shifts theposition of the window area, to thereby obtain the motion vector foreach pixel or each block (for example, 8×8 pixels).

The W pixels having a larger number of pixels and a higher sensitivitythan the RGB pixels are used for motion detection, to thereby be able toincrease a spatial resolution of the motion vector and increase atolerance to noise in the calculation of the motion vector.

When the absolute value of the motion vector exceeds the threshold valueVth, the discrimination processing unit 213 sets the discriminationsignal J to “1”. In this case, the inter-frame processing unit 214reduces the number of frames to be subjected to the inter-frameprocessing for the color signal, or inhibits the inter-frame processingfrom being conducted. With this operation, it is possible to suppressthe color afterimage caused by conducting the color signal processingfor a part exhibiting a motion. Meanwhile, when the absolute value ofthe motion vector is equal to or smaller than the threshold value Vth,the discrimination signal J is “0”, and the inter-frame processing unit214 conducts the inter-frame processing for the color signal, andconducts the noise reduction for the color signal.

This embodiment can also produce the same effects as those of the firstand second embodiments. In addition, in this embodiment, it isdetermined whether or not to conduct the inter-frame processing for thecolor signal depending on a magnitude of the motion vector within the Wpixels. The W pixels have a high spatial resolution and a highsensitivity, and hence the discrimination of the inter-frame processingcan be conducted with high accuracy. As a result, it is possible toobtain an effect of the noise reduction for the color signal whilefurther reducing the color afterimages.

Fourth Embodiment

FIG. 15 is a block diagram of the signal processing unit 2 of an imagingapparatus according to a fourth embodiment of the present invention. Theimaging apparatus according to this embodiment is different from theimaging apparatus according to the first embodiment in that the RGBW8array illustrated in FIG. 4C is used. The imaging apparatus according tothe fourth embodiment is described below mainly in terms of pointsdifferent from those of the first embodiment.

In the RGBW8 array, the number of W pixels is smaller than that of theRGBW12 array, and hence the sensitivity is lower. Meanwhile, RGB pixelsexist around W pixels, and hence the false color is less liable tooccur. In FIG. 15, a pixel signal 4 a of the CF array of RGBW8 isseparated into a pixel signal 4 b of W being a luminance signal and apixel signal 4 e of RGB being a color signal.

The luminance signal processing unit 204 obtains a pixel value in eachof parts from which the RGB pixels have been separated within the pixelsignal 4 b by the interpolation processing, and generates a pixel signal4 c subjected to the interpolation. The interpolated pixels arerepresented by “iWr”, “iWg”, and “iWb”.

The control unit 205 includes the spatial average processing unit 212and the discrimination processing unit 213. The spatial averageprocessing unit 212 is configured to calculate the average value W_(ave)of the pixel values for a predetermined block within the pixel signal 4b of W. The discrimination processing unit 213 is configured to comparethe inter-frame differential of the average value W_(ave) with thethreshold value Vth. When the absolute value of the inter-framedifferential is larger than the threshold value Vth, the discriminationsignal J is “1”, and when the absolute value of the inter-framedifferential is equal to or smaller than the threshold value Vth, thediscrimination signal J is “0”.|W _(ave)(N)−W _(ave)(N−1)|>Vth→J=1|W _(ave)(N)−W _(ave)(N−1)|≦Vth→J=0

The color signal processing unit 206 includes the inter-frame processingunit 214. The processing of the inter-frame processing unit 214 isconfigured in the same manner as in the first embodiment. That is, whenthe discrimination signal J is “1”, the inter-frame processing unit 214reduces the number of frames to be subjected to the inter-frameprocessing for the color signal, or inhibits the inter-frame processingfrom being conducted. That is, when the temporal change of the pixelvalue of W is large, it is possible to reduce the color afterimagescaused by the object exhibiting a motion by reducing the number offrames to be subjected to the inter-frame processing for the colorsignal.

The signal combining unit 207 combines the pixel signal 4 c of Wsubjected to the interpolation and a pixel signal of RGB subjected tothe color signal processing. That is, the signal combining unit 207calculates the color ratio of the pixel values of W and iW and the pixelvalues of n_R, n_G, and n_B in the same position, and multiplies thecolor ratio by the pixel values of W and iW, to thereby calculate thevalue of RGB for each pixel. In the same manner as in the firstembodiment, the pixel value of RGB is obtained by one of the followingexpressions for each of the pixels of W and iW.

${RGB} = \begin{bmatrix}{\frac{n\_ R}{iWr}W} & {\frac{n\_ G}{iWg}W} & {\frac{n\_ B}{iWb}W}\end{bmatrix}$ ${RGB} = \begin{bmatrix}{\frac{n\_ R}{iWr}{iW}} & {\frac{n\_ G}{iWg}{iW}} & {\frac{n\_ B}{iWb}{iW}}\end{bmatrix}$

In this embodiment, through the use of the RGBW8 array, the sensitivityand the resolution of an image became lower than the first embodiment,but the reduction in the false colors was enabled depending on thedesign pattern of an object.

Fifth Embodiment

FIG. 16 is a block diagram of the signal processing unit 2 of an imagingapparatus according to a fifth embodiment of the present invention. Theimaging apparatus according to this embodiment is different from theimaging apparatus according to the first embodiment in that the RGBG12array illustrated in FIG. 4D is used. The imaging apparatus according tothe fifth embodiment is described below mainly in terms of pointsdifferent from those of the first embodiment.

In the RGBG12 array, the W pixel of RGBW12 is replaced by the G pixel,and hence the sensitivity is liable to be lowered. However, the W pixelexhibits a higher sensitivity than the RGB pixel, and hence, when animage of the object having a high luminance is picked up, the W pixelcan be saturated, and the dynamic range can be lowered. In thisembodiment, through the use of the CFs of the RGBG12 array, thesensitivity and the saturation of the signal can be balanced.

In FIG. 16, the pixel signal 5 a is separated into a pixel signal 5 b ofG and a pixel signal 5 e of R and B.

The luminance signal processing unit 204 conducts interpolationprocessing for parts from which the pixels of R and B have beenseparated within the pixel signal 5 b to generate pixel values of iGrand iGb. The spatial average processing unit 212 calculates the averagevalue W_(ave) of the pixel values of G within a predetermined block ofthe pixel signal 5 b of G. The discrimination processing unit 213compares the inter-frame differential of the average value W_(ave) withthe threshold value Vth. When the absolute value of the inter-framedifferential is larger than the threshold value Vth, the discriminationsignal J is “1”, and when the absolute value of the inter-framedifferential is equal to or smaller than the threshold value Vth, thediscrimination signal J is “0”.|G _(ave)(N)−G _(ave)(N−1)|>Vth→J=1|G _(ave)(N)−G _(ave)(N−1)|≦Vth→J=0

The sensitivity of the G pixel is lower than the sensitivity of the Wpixel, but the G pixel includes a larger amount of luminance informationthan the pixels of R and B. When an object moves, not only the color butalso the luminance often changes. Therefore, the inter-framedifferential of the G pixel is obtained, to thereby be able to increasethe accuracy in the discrimination of the inter-frame processing.

The color signal processing unit 206 conducts the inter-frame processingfor a pixel signal 5 e of R and B. The processing of the inter-frameprocessing unit 214 is the same as that of the first embodiment. Whenthe discrimination signal J is “1”, the inter-frame processing unit 214reduces the number of frames to be subjected to the inter-frameprocessing to a smaller number than when the discrimination signal J is“0”, or inhibits the inter-frame processing from being conducted. Thatis, when the temporal change of the pixel value of G is large, thenumber of frames to be subjected to the inter-frame processing for thecolor signal is reduced. With this operation, it is possible to reducethe colored afterimages caused by the object exhibiting a motion.

The signal combining unit 207 combines a pixel signal 5 c of G subjectedto the interpolation and the inter-frame processed pixel signals of Rand B. That is, the signal combining unit 207 calculates the color ratioof the pixel values of G and iG and the pixel values of n_R and n_B inthe same position, and multiplies the color ratio by the pixel values ofG and iG, to thereby calculate the value of RGB for each pixel. Thepixel value of RGB is obtained by one of the following expressions foreach of the pixels of G and iG.

${RGB} = \begin{bmatrix}{\frac{n\_ R}{iWr}W} & {\frac{n\_ G}{iWg}W} & {\frac{n\_ B}{iWb}W}\end{bmatrix}$ ${RGB} = \begin{bmatrix}{\frac{n\_ R}{iWr}{iW}} & {\frac{n\_ G}{iWg}{iW}} & {\frac{n\_ B}{iWb}{iW}}\end{bmatrix}$

In a photographed image, the sensitivity and the resolution were lowerthan in the first embodiment, but through use of RGB pixels, thereduction in the false colors caused when a moving image was beingphotographed was enabled while the saturation was suppressed. In thismanner, the luminance signal is not limited to the signal of the W pixelunlike in the first embodiment, and it suffices that the luminancesignal is information of a pixel including a large amount of luminanceinformation (for example, G pixel) in a visual characteristic. Further,it suffices that the color signal is the signal of a pixel including arelatively small amount of luminance information (for example, R pixeland B pixel). In addition, in this embodiment, the pixel signal 5 a isseparated into the pixel signal 5 b of G and the pixel signal 5 e of Rand B, but the same effects can be produced also by separating the dataincluding a large amount of luminance information and the data includinga small amount of luminance information through an arithmetic operation.

Sixth Embodiment

FIG. 17 is a block diagram of the signal processing unit 2 of an imagingapparatus according to a sixth embodiment of the present invention.

The imaging apparatus according to this embodiment is described belowmainly in terms of points different from those of the first embodiment.In this embodiment, the imaging device 1 uses a CMYW12 array illustratedin FIG. 5B. The CMYW12 array uses the W pixels in addition to the pixelsof complementary colors (C, M, and Y) having a high sensitivity, tothereby be able to improve the sensitivity.

In FIG. 17, a pixel signal 6 a received from the imaging device 1 isseparated into a pixel signal 6 b of W and pixel signals 6 e of C, M,and Y.

The luminance signal processing unit 204 conducts interpolationprocessing for parts from which the pixels of C, M, and Y have beenseparated within the pixel signal 6 b to generate the pixel values ofiWc, iWm, and iWy. The spatial average processing unit 212 of thecontrol unit 205 calculates the average value W_(ave) of the pixelvalues within a predetermined block of the pixel signal 6 b of W. Thediscrimination processing unit 213 compares the inter-frame differentialof the average value W_(ave) with the threshold value Vth. When theabsolute value of the inter-frame differential is larger than thethreshold value Vth, the discrimination signal J is “1”, and when theabsolute value of the inter-frame differential is equal to or smallerthan the threshold value Vth, the discrimination signal J is “0”.|W _(ave)(N)−W _(ave)(N−1)|>Vth→J=1|W _(ave)(N)−W _(ave)(N−1)|≦Vth→J=0

The inter-frame processing unit 214 of the color signal processing unit206 conducts the inter-frame processing for a pixel signal of CMY beinga second pixel group. When the discrimination signal J is “1”, theinter-frame processing unit 214 reduces the number of frames to besubjected to the inter-frame processing to a smaller number than whenthe discrimination signal J is “0”, or inhibits the inter-frameprocessing from being conducted. With this operation, it is possible toreduce the colored afterimages caused by a motion of the objectexhibiting the motion. The signal combining unit 207 combines a pixelsignal 6 c of W subjected to the interpolation and the pixel signal ofCMY subjected to the color signal processing. That is, the signalcombining unit 207 calculates of the color ratio of the pixel values ofW and iW and the pixel values of n_C, n_M, and n_Y in the same position,and multiplies the color ratio by the pixel values of W and iW, tothereby calculate the value of CMY for each pixel. The pixel value ofCMY is obtained by one of the following expressions for each of thepixels of W and iW.

${CMY} = \begin{bmatrix}{\frac{n\_ C}{iWc}W} & {\frac{n\_ M}{iWm}W} & {\frac{n\_ Y}{iWy}W}\end{bmatrix}$ ${CMY} = \begin{bmatrix}{\frac{n\_ C}{iWc}{iW}} & {\frac{n\_ M}{iWm}{iW}} & {\frac{n\_ Y}{iWy}{iW}}\end{bmatrix}$

A CMY/RGB converting unit 287 converts the pixel values of CMY outputfrom the signal combining unit 207 into the pixel value of RGB, andoutputs an image signal 6 g. The imaging apparatus used to conduct theabove-mentioned processing was used to conduct evaluation photographing.The sensitivity was higher than in the imaging apparatus according tothe first embodiment even though color reproducibility was lowerpartially in an image pattern, and the false color caused when a movingimage was being photographed was suppressed. The processing of thesignal combining unit 207 may be executed after the processing of theCMY/RGB converting unit 287, or the two pieces of processing may beexecuted integrally.

Seventh Embodiment

FIG. 18 is a block diagram of the signal processing unit 2 of an imagingapparatus according to a seventh embodiment of the present invention.The imaging apparatus according to this embodiment is described belowmainly in terms of points different from those of the first embodiment.In this embodiment, the imaging device 1 uses the Bayer (RGB) arrayillustrated in FIG. 4A. The luminance signal processing unit 204conducts processing through use of the pixel value of G as the luminancesignal, and the color signal processing unit 206 conducts processingthrough use of the pixel values of R and B as color signals. The Bayerarray has a characteristic of having a lower sensitivity than the CFsusing W pixels. However, according to this embodiment, the pixel valuesof R and B having relatively low sensitivity are subjected to theinter-frame processing, to thereby be able to suppress the color noisecaused when the illuminance is low. Further, the discrimination of theinter-frame processing is conducted through use of the G pixel having ahigher sensitivity and a higher resolution than the pixels of R and B,to thereby be able to improve the accuracy in the discrimination andreduce the color afterimages.

In FIG. 18, a pixel signal 7 a of the Bayer (RGB) array is separatedinto a pixel signal 7 b of G and a pixel signal 7 e of RB.

The luminance signal processing unit 204 conducts interpolationprocessing for parts from which the pixels of R and B have beenseparated to generate the pixel values of iGr and iGb. The spatialaverage processing unit 212 of the control unit 205 calculates anaverage value G_(ave) of the pixel values within a predetermined blockof the pixel signal 7 b of G. The discrimination processing unit 213obtains the inter-frame differential for the spatial average value ofthe G pixel, and compares the inter-frame differential with thethreshold value Vth. When the absolute value of the inter-framedifferential is larger than the threshold value Vth, the discriminationsignal J is “1”, and when the absolute value of the inter-framedifferential is equal to or smaller than the threshold value Vth, thediscrimination signal J is “0”.|G _(ave)(N)−G _(ave)(N−1)|>Vth→J=1|G _(ave)(N)−G _(ave)(N−1)|≦Vth→J=0

The color signal processing unit 206 conducts the inter-frame processingfor the RB pixel signal 7 e of RB. That is, when the discriminationsignal J is “1”, the inter-frame processing unit 214 reduces the numberof frames to be subjected to the inter-frame processing to a smallernumber than when the discrimination signal J is “0”, or inhibits theinter-frame processing from being conducted. That is, when the temporalchange of the pixel value of G is large, the number of frames to besubjected to the inter-frame processing for the color signal is reduced.With this operation, it is possible to reduce the color afterimagescaused by the object exhibiting a motion.

The signal combining unit 207 combines a pixel signal 7 c of G subjectedto the interpolation and the inter-frame processed pixel signals of RB.The signal combining unit 207 calculates the color ratio of the pixelvalues of G and iG and the pixel values of n_R and n_B in the sameposition, and multiplies the color ratio by the pixel values of G andiG. Therefore, the value of RGB is calculated for each pixel, and animage signal 7 g is generated. The image value of RGB is obtained by oneof the following expressions for each of the pixels of G and iG.

${RGB} = \begin{bmatrix}{\frac{n\_ R}{iGr}G} & G & {\frac{n\_ B}{iGb}G}\end{bmatrix}$ ${RGB} = \begin{bmatrix}{\frac{n\_ R}{iGr}{iG}} & {iG} & {\frac{n\_ B}{iGb}{iG}}\end{bmatrix}$

In a photographed image, both the reduction in color noise and thereduction in color afterimages can be achieved with a low illuminancethrough the use of the CFs of the Bayer array.

Eighth Embodiment

An imaging system according to an eighth embodiment of the presentinvention is described. The imaging apparatus according to theabove-mentioned first to seventh embodiments can be applied to variousimaging systems. The imaging system is an apparatus configured toacquire an image, a moving image, and the like through use of theimaging apparatus, and examples thereof include a digital still camera,a digital camcorder, and a surveillance camera. FIG. 19 is a blockdiagram for illustrating a system in which the imaging apparatusaccording to one of the first to seventh embodiments is applied to adigital still camera employed as an example of the imaging system.

In FIG. 19, the imaging system includes a lens 302 configured to imagean optical image of an object on an imaging apparatus 301, a barrier 303for protection of the lens 302, and a diaphragm 304 for adjustment of anamount of light that has passed through the lens 302. The imaging systemincludes an output signal processing unit 305 configured to process anoutput signal output from the imaging apparatus 301.

The output signal processing unit 305 includes a digital signalprocessing unit, and is further configured to conduct an operation ofsubjecting the signal output from the imaging apparatus 301 to variouskinds of correction and compression as the need arises, and outputtingthe signal. When the signal output from the imaging apparatus 301 is ananalog signal, the output signal processing unit 305 may include ananalog-to-digital conversion circuit in the previous stage of thedigital signal processing unit.

The imaging system includes a buffer memory unit 306 for temporarilystoring image data and a recording medium control interface (I/F) unit307 for conducting recording or reading into or from the recordingmedium. The imaging system further includes a recording medium 309 forrecording or reading the image data, such as a semiconductor memory,which can be inserted into or removed from the imaging system or isbuilt into the imaging system. The imaging system further includes anexternal interface (I/F) unit 308 for communicating to or from anexternal computer or the like and a general control/operation unit 310configured to conducting various arithmetic operations and overallcontrol of the digital still camera. The imaging system further includesa timing generation unit 311 configured to output various timing signalsto the output signal processing unit 305. A control signal such as atiming signal may be input from the outside instead of from the timinggeneration unit 311. That is, it suffices that the imaging systemincludes at least the imaging apparatus 301 and the output signalprocessing unit 305 configured to process an output signal output fromthe imaging apparatus 301.

As described above, the imaging system according to this embodiment canconduct an imaging operation through application of the imagingapparatus 301 described in the first to seventh embodiments.

Other Embodiments

While an imaging apparatus in the present invention has been described,the present invention is not limited to the embodiments given above, andthe embodiments are not to inhibit suitable modifications and variationsthat fit the spirit of the present invention. For example, theconfigurations of the above-mentioned first to eighth embodiments canalso be combined. The imaging apparatus does not necessarily include animaging device, and may be an image processing system such as a computerconfigured to process an image signal output from the imaging device.

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executescomputer-executable instructions (e.g., one or more programs) recordedon a storage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer-executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer-executable instructions. The computer-executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2015-101706, filed May 19, 2015, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An imaging apparatus, comprising: an imagingdevice; and a signal processing unit, wherein: the imaging devicecomprises a first pixel group and a second pixel group each comprising aplurality of pixels each configured to output a pixel signal; the pixelsignal output by the plurality of pixels of the second pixel groupincludes a smaller amount of resolution information than an amount ofresolution information included in the pixel signal output by theplurality of pixels of the first pixel group; and the signal processingunit is configured to perform weighted addition for a second pixelsignal output from the second pixel group by inter-frame processing, andto change a weight on each frame in the weighted addition based on aninter-frame differential of a first pixel signal.
 2. An imagingapparatus according to claim 1, further comprising a discriminationprocessing unit configured to output a discrimination signal indicatinga result of comparing the inter-frame differential of the first pixelsignal with a threshold value, wherein the signal processing unit isfurther configured to change the weight on each frame based on a signalvalue of the discrimination signal.
 3. An imaging apparatus according toclaim 2, wherein the discrimination processing unit is furtherconfigured to calculate an average value of signals output from a largernumber of first pixels than a number of second pixels to be subjected tothe inter-frame processing, and to generate the discrimination signalbased on an inter-frame differential of the average value.
 4. An imagingapparatus according to claim 2, wherein the signal processing unit isfurther configured to reduce a weight on another frame of the secondsignal with respect to a weight on a current frame of the second signalas the inter-frame differential of the first pixel signal becomeslarger.
 5. An imaging apparatus according to claim 3, wherein the firstpixels to be subjected to calculation of the average value comprisefirst pixels around the second pixel to be subjected to the inter-frameprocessing.
 6. An imaging apparatus according to claim 2, wherein: thediscrimination processing unit is further configured to generate thediscrimination signal based on an inter-frame differential ofinformation of a ratio between a first pixel signal interpolated in aposition of a second pixel and the second pixel signal; and the signalprocessing unit is further configured to reduce a weight on anotherframe of the second signal with respect to a weight on a current frameof the second signal as the inter-frame differential of the informationof the ratio becomes larger.
 7. An imaging apparatus according to claim2, wherein: the discrimination processing unit is further configured tooutput the discrimination signal by calculating a motion vector throughuse of a plurality of frames of the first pixel signal; and the signalprocessing unit is further configured to reduce a weight on anotherframe of the second signal with respect to a weight on a current frameof the second signal as an absolute value of the motion vector becomeslarger.
 8. An imaging apparatus according to claim 1, further comprisinga signal combining unit configured to combine the first pixel signal andthe second pixel signal subjected to the inter-frame processing togenerate an image signal obtained by expressing each pixel by respectivevalues of R, G, and B.
 9. An imaging apparatus according to claim 1,wherein the inter-frame processing comprises a moving average.
 10. Animaging apparatus according to claim 1, wherein the inter-frameprocessing is conducted by a recursive filter.
 11. An imaging apparatusaccording to claim 1, wherein the inter-frame processing is conducted bya non-recursive filter.
 12. An imaging apparatus according to claim 1,wherein the inter-frame processing is conducted by a median filter. 13.An imaging apparatus according to claim 1, wherein the first pixel groupis formed of W pixels.
 14. An imaging apparatus according to claim 1,wherein the second pixel group is formed of an R pixel, a G pixel, and aB pixel.
 15. An imaging apparatus according to claim 1, wherein a numberof pixels of the first pixel group is larger than a number of pixels ofthe second pixel group.
 16. An imaging apparatus according to claim 1,wherein a pixel of the second pixel group is surrounded by pixels of thefirst pixel group.
 17. An imaging system, comprising: an imagingapparatus; and an output signal processing unit configured to process asignal output from the imaging apparatus, the imaging apparatuscomprising: an imaging device; and a signal processing unit, wherein:the imaging device comprises a first pixel group and a second pixelgroup each comprising a plurality of pixels each configured to output apixel signal; the pixel signal output by the plurality of pixels of thesecond pixel group includes a smaller amount of resolution informationthan an amount of resolution information included in the pixel signaloutput by the plurality of pixels of the first pixel group; and thesignal processing unit is configured to perform weighted addition for asecond pixel signal output from the second pixel group by inter-frameprocessing, and to change a weight on each frame in the weightedaddition based on an inter-frame differential of a first pixel signal.18. An image processing method for processing a pixel signal output froman imaging apparatus, the imaging apparatus comprising: an imagingdevice; and a signal processing unit, the imaging device comprising afirst pixel group and a second pixel group each comprising a pluralityof pixels each configured to output the pixel signal, the pixel signaloutput by the plurality of pixels of the second pixel group including asmaller amount of resolution information than an amount of resolutioninformation included in the pixel signal output by the plurality ofpixels of the first pixel group, the image processing method comprisingperforming weighted addition for a second pixel signal output from thesecond pixel group by inter-frame processing, and changing a weight oneach frame in the weighted addition based on an inter-frame differentialof a first pixel signal.
 19. An imaging device, comprising: a firstpixel group and a second pixel group each comprising a plurality ofpixels each configured to output a pixel signal; and a signal processingunit, wherein: the pixel signal output by the plurality of pixels of thesecond pixel group includes a smaller amount of resolution informationthan an amount of resolution information included in the pixel signaloutput by the plurality of pixels of the first pixel group; and thesignal processing unit is configured to perform weighted addition for asecond pixel signal output from the second pixel group by inter-frameprocessing, and to change a weight on each frame in the weightedaddition based on an inter-frame differential of a first pixel signal.20. An imaging device according to claim 19, wherein: the first pixelgroup and the second pixel group are formed on a first semiconductorsubstrate; the signal processing unit is formed on a secondsemiconductor substrate; and the first semiconductor substrate and thesecond semiconductor substrate are laminated on one another.